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Abstract. We prove that a well-known conjecture of Margulis 
on the action of the diagonal group A on the space of unimodular 
lattices, is equivalent to several assertions about minimal sets for 
this action. More generally, for a maximal R-diagonalizable sub- 
group A of a reductive group G and a lattice V in G, we give a 
sufficient condition for a compact A- minimal subset Y of G/T to 
be of a simple form, which is also necessary if G is R-split. We also 
show that the stabilizer of Y has no nontrivial connected unipotent 
subgroups. 



1. Introduction 

Given a group A acting on a locally compact space X, we say that 
the action is minimal if every A-orbit is dense, or equivalently, if there 
are no proper A-invariant closed subsets. We say that a subset Y C X 
is a minimal set if it is A-invariant and closed, and the restriction of the 
A-action to Y is minimal. If X is compact then any closed invariant 
subset contains a minimal set, and the study of minimal sets is often 
an important first step in the study of all A-orbits in X. The study of 
minimal sets may be viewed as the topological counterpart of the study 
of invariant ergodic measures. In many dynamical systems, there are 
many minimal sets defying a simple classification; however, as we will 
see, in some dynamical systems of algebraic origin, the classification of 
minimal sets might be possible and is of great interest. 

This paper is devoted to the study of minimal sets for the action 
of A on G/T, where G is a connected reductive real algebraic group, 
T is a lattice (i.e. a discrete subgroup such that the quotient G/T 
supports a finite G-invariant measure), and A is a maximal connected 
IR-diagonlizable subgroup of G. Recall that a root on A is a nontrivial 
homomorphism a : A — > M* such that there is a nonzero v G g = 
Lie(G) such that for any a G A, Ad(a)v = a(a)v, where M* denotes 
the multiplicative group of positive real numbers. We will denote the 
kernel of a by A a ; this is a subgroup of A which is of codimension one. 

l 
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We will pay particular attention to the action of the groups A a on a 
minimal set for A. 

We begin with the important special case in which G = SL n (R), n > 
3, T = SL n (Z), and A is the subgroup of all diagonal matrices with pos- 
itive diagonal entries. In this case the space G/Y is naturally identified 
with the space of unimodular lattices in W 1 , where the action is simply 
the linear action of a matrix as a linear isomorphism of W 1 . The dy- 
namics of the A-action on G/Y has been intensively studied, both for 
its intrinsic interest as a prototypical action for which one may hope to 
classify all minimal sets and invariant measures, and for its connections 
to number-theoretic questions. See |ELj for a survey of the history as 
well as many recent developments. Several questions about this action 
were raised in 1955 by Cassels and Swinnerton-Dyer |CaSD] , which 
are open to this day, and several conjectures were made in 2000 by 
Margulis in |Ma2] . One of them is the following: 

Conjecture 1 (Margulis). Let n > 3, G = SL n (R), Y = SL n (Z), and 
let A be the group of positive diagonal matrices. Any A-orbit on G/Y 
which is bounded (i.e. has compact closure) is closed (and hence com- 
pact). 

Following ideas already presented in |CaSD] , Margulis |Mal] showed 
that Conjecture [1] implies a well-known number-theoretic conjecture 
of Littlewood. In §[2] we will show that Conjecture [1] is equivalent to 
statements regarding minimal sets for the A-action. Namely we will 
show: 

Theorem 2. Let G,Y,A be as in ConjectureYJl The following asser- 
tions are equivalent to ConjectureY^ 

(a) Any compact minimal set for the A-action on G/Y is a compact 
orbit. 

(b) For any root a and any compact A-minimal set Y C G/Y, Y is 
also minimal for the action of A a . 

(c) For any root a and any compact minimal set Y C G/Y for the 
action of A, any A a -minimal subset Y a C Y satisfies AY a = Y . 

(d) For any root a and any compact minimal set Y C G/Y for the 
action of A, there exists an A a -minimal subset Y a C Y such 
that AY a = Y. 

Now we turn to the general case. The assertion of Conjectured] is not 
true for general G/Y. In fact, in unpublished work, Mary Rees showed 
that it does not hold even for G = SL^R) and Y a certain cocompact 
lattice. An interesting feature of Rees' examples is that the groups A a 
do not act minimally on A-minimal sets, see the discussion in \YW\ §5]. 
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That is, Rees' example also shows that condition (b) of Theorem |2] need 
not hold in general. In this paper we clarify the implications among 
the conditions (a-d), in the context of a general G/T; among other 
things, it will develop that it is no coincidence, that for certain G/T, 
the same constructions show the failure of Conjecture [1] and condition 
(b). More precisely, we will prove the following result. 

Theorem 3. Let G be a connected reductive real algebraic group, T 
a lattice in G, A a maximal connected M,-diagonlizable subgroup of G, 
and Y C G/T a compact A-minimal set. Suppose that for any root a, 
there exists an A a -minimal set Y a cY such that AY a = Y . Then there 
exists a compact subtorus T of G centralized by A such that Y is an 
AT -orbit. 

Note that if G is R-split, then A is of finite index in Zg{A), and 
there is no nontrivial compact torus centralized by A. So we have the 
following corollary, which implies the equivalence of (a), (c) and (d) in 
Theorem [2J 

Corollary 4. Under the conditions of Theorem^ suppose in addition 
that G is WL-split, then the following are equivalent: 

(a) Y is a single (compact) A-orbit. 

(b) For any root a, any A a -minimal set Y a <zY satisfies AY a = Y . 

(c) For any root a, there exists an A a -minimal set Y a C Y such 
that AY a = Y. 

For a Lie group H , we denote its connected component of the identity 
by H°. For any compact A-minimal set Y C G/T, denote 

St a b G (Y) d ^{geG:gY = Y}. 

The proof of Theorem [3] is based on the following result, which is of 
interest in itself. 

Theorem 5. Let G, T,A be as in Theorem^ and let Y C G/T be a 
compact A-minimal set. Then StabctX) ^ as no nontrivial connected 
unipotent subgroups. Equivalently, we have Stabc(^) C Zq{A). 

If G is R-split, Theorem El asserts that Stabc(^) = A, i.e., A is of 
finite index in Stabc(y). 

The proof of Theorem relies on work of Dani and Margulis [DM] 
and Prasad and Raghunathan |PRj . It also relies on a statement 
(Proposition [8] below) about orbit-closures of elements of an A-minimal 
set, under a unipotent group normalized by A. The proof of Proposi- 
tion [8] employs some ideas of Mozes [Mj. Throughout this paper, by 
a real algebraic group, we mean an open subgroup of the Lie group of 
real points of an algebraic group defined over R. 
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We begin with a simple proof of Theorem [21 based on the results of 
[LW] . We will need the following: 

Proposition 6. Let G/T = SL n (R)/ SL„(Z), let A be the group of diag- 
onal positive matrices, and let U be a one-parameter unipotent subgroup 
of G normalized by A. Then any AU -orbit in G/T is unbounded. 

Proof. This was proved in |CaSD] and again in |LW] ; we repeat the 
proof for completeness. 

Recall Mahler's criterion, which asserts that X d G /V is bounded if 
and only if there is e > such that any nonzero vector in any lattice in 
X has length at least e. Since A normalizes U, there are distinct i,j G 
{1, . . . ,n} such that, if we define u(s) = exp(sEij), then U = {u(s) : 
s G K}, where Eij is the matrix with 1 in the (i,j)th entry and zero 
elsewhere. To simplify notation assume j — 1. Let x G G/T, and let 
v — (v\, . . . , v n ) be a nonzero vector in the lattice corresponding to x. If 

v x — then we can apply the elements a(t) = f diag(e^ n_1 ^, e~ l , . . . , e~ l ), 
and we will have a(t)v -^-t^oo 0, implying via Mahler's compactness 
criterion that Ax is unbounded. If v± ^ then we can apply u(—Vi/v\) 
to x to obtain, in the lattice corresponding to u(— Vi/v\)x, a vector 
with vanishing i-th entry. Now we repeat the previous step to this 
vector. □ 

Proof of Theorem^ Conjectured] <^=^> (a): Suppose Conjecture [1] 
holds. Any orbit inside any compact minimal set Y is bounded, hence 
a compact orbit, which by minimality coincides with Y. Conversely, 

assume (a), and suppose Ax is a bounded orbit. Then Xq = f Ax con- 
tains a minimal set and hence a compact orbit. Now an argument of 
[CaSD] (see also |Mal| ILW] ) shows that if X is not itself a compact 
A-orbit, then it contains an AU-orbit and hence cannot be compact, 
in view of Proposition [6j 



2. Conjectures for the A-action 
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(a) ==>- (b): This is proved in |LWl Step 6.1]. 

(b) ==>- (c) ==>- (d): Immediate. 

(d) ==>- (c): Suppose Y a is minimal with AY a = Y. Then the sets 
{aY a : a G A} are all A a -mmimal, and they cover Y. Thus if Y' C Y is 
another v4 a -minimal set, then there is a G A such that Y' = aY a , and 
hence AY' = Y. 

(c) ==>- (a): If (a) is false then there is an A- minimal set Y which 
does not contain a compact A-orbit. Let oq G A be a regular element, 
i.e. a G A \ |J A a . For any root a, let u a be a generator of the root 
space 

u a = f {v G g : Va G A, Ad(a)v = a(a)v}, 

and let u a (s) = exp(su Q ), U a = {u a (s) : s G K}. Let £/ + be the group 
generated by {f/ Q : a(a ) > 0}. Arguing as in \UW\ Steps 4.3, 4.4] (up 
to replacing a by a^ 1 ), we see that there are distinct x±,x 2 G Y and 
u + G t/ + \ {e} such that x 2 = u + x\. Arguing as in \UW\ Step 4.5], 
there is a root a, distinct yi,y2 G Y and u = u a (s ) G U a , s ^ such 
that ?/2 = wyi. In light of Proposition El it suffices to show that Y 
contains a C/ a -orbit. 

Since AY a = Y, and each aY a is A a -minimal, we can assume that 

Hi G F a , and we claim that y 2 G Y a . If not then y 2 G F' = f a'Y" Q for 
some a'6i \ A a , where y' is also Aa-minimal, and there is a positive 
distance between Y' and Y a . According to Lemma [10l which we prove 
below in a more general setting, there is a G A\A a such that aY a = Y a . 
Replacing a with a" 1 if necessary, we may assume that a(a) < 1. Then 
for each fceN, 

y' 9 a fc ?/2 = aku Ui = exp(a(a) fc s u Q ,)a fc i/i G exp(a(a fc )soU Q ,)y a . 

Since a(a) k — >k^-oo 0, the distance from Y' to Y a is bounded above by a 
quantity tending to zero as k -> oo, a contradiction proving the claim. 
Since y 2 = uyi G Y a , and the y4 a -action commutes with that of u 

we find that H = f Stabc^a) contains w. Since if is a closed group, it 
contains all conjugates of u by a and hence contains U a . This completes 
the proof. □ 

3. Proof of Theorem [5] 

We will need the following statement, which is proved in |SWj in a 
more restricted setting: 

Proposition 7. Let G be a real algebraic group, T a lattice in G, L 
a connected reductive algebraic subgroup of G, and y G G/Y. Suppose 
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that the orbit Ly is closed. Then L can be decomposed as a direct 
product L = Z x U of its connected closed normal subgroups such that 

(i) L' contains Lj, = {£ G L : ly = y} as a lattice, and contains the 
maximal connected normal subgroup of G with compact center. 

(ii) Z is an M.-diagonalizable algebraic subgroup of the center of L. 

(iii) The natural map 

Z x (V ' /L y ) ->■ Ly, {z,£L y ) !-)■ ziy 

is a homeomorphism. 

Proof. Without loss of generality, we may assume that y is the pro- 
jection of e in G/T, and hence L y = L D T. Let S be the maximal 
connected normal semisimple subgroup of L without compact factors. 
By Ratner's theorem, there is a connected closed subgroup S' of G 
containing S such that Sy = S'y and 5" D T is a lattice in 5". Since Ly 
is closed, we have S' C L. Let K be the maximal connected normal 
compact subgroup of L, and let N = KS' . Then N is a connected 
closed normal subgroup of L and L/N is a simply connected abelian 
Lie group. Since S'y is closed in Ly, S'L y , and hence NL y , is closed in 
L. Thus q(L y ) is a discrete subgroup of L/N, where q : L — > L/N is 
the quotient homomorphism. Let C be the unique connected subgroup 
of L/N which contains q(L y ) as a lattice, and let V = q~ l (C). We 
prove that V satisfies (i). It suffices to prove that L y is a lattice in 
L' . Firstly, since q{L y ) is a lattice in C, V /NL y carries an //-invariant 
finite measure. On the other hand, since S' fl V is a lattice in S' , it 
follows that N H T is a lattice in N, and hence L y is a lattice in NL y 



(see Rag Lem. 1.7]). Thus L y is a lattice in L' (see |Rag Lem. 1.6]). 

Since L' satisfies (i), we can choose Z satisfying (ii) such that L = 
Z x V . Moreover, the natural map in (iii) is the composition of the 
natural homeomorphisms Z x (L'/L y ) = L/L y and L/L y = Ly, hence 
is a homeomorphism. This completes the proof. □ 

In the rest of this paper, we assume that G, T, A are as in Theorem [31 
i.e., G is a connected reductive real algebraic group, r C G is a lattice, 
and A C G is a maximal connected IR-diagonlizable subgroup. Let $ = 
$(A,G) be the root system. For a G $, let u a C g be the root space 
of a, and let ?7 a = exp(u Q ) be the root group. The following statement 
will be important for the sequel, and is of independent interest. Its 
proof is inspired by ideas of Shahar Mozes [M] • 



Proposition 8. Let Y C G/T be a compact A-minimal set, U be a 
connected unipotent subgroup of G normalized by A. Then there exist 
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an A-invariant residual subset Yq C Y and a connected reductive closed 
subgroup H of G which contains U and is normalized by A such that 

(i) For every y £ Yq, we have Uy = Hy. 

(ii) The algebraic subgroup L d = Ng{H)° of G is reductive, and Y 
is contained in a closed L-orbit of finite volume. 

Proof. We divide the proof into steps. 

Step 1. Let 7r : G — > G/Y be the natural projection, and let % be 
the collection of connected closed subgroups H of G for which Hir(e) = 
H/Y n H is closed and of finite volume, and such that the subgroup of 
H generated by its unipotent elements acts ergodically on Hir(e). It 
was shown by Ratner (see [R]) that H is countable, and for any g £ G, 
there is H £ Ti such that U7i(g) = (gHg~ 1 )7r(g). For any H £ %, let 

X(H,U) d ^{geG:UgCgH}. 

Each X(H, U) is an algebraic subvariety of G, and for any g £ X(H, U) 
we have 

TM^CgHn(e) = (gHg~ 1 )n(g). 

Note that 

N G (U)X(H, U)N G (H) = X(H, U). 

In particular, X(H, U) is left A-invariant. Note also that G £ %, and 
thus G/Y = \J H£H ir(X(H, U)). 

Let H £ % be a group of minimal dimension for which tt(X(H , U)) 
contains an open subset of Y. Since Y is A-minimal and tt(X(Ho, U)) 
is v4-invariant, we have 

Y Ctt(X(H ,U)). 

By the choice of H , if H £ H is a proper subgroup of H , then 
Y s ir(X(H, U)) is a residual subset of F, and hence so is 

|J 7T(X(/f,f/)). 
H£H,H^H 

Note that Yq is A-invariant. It is easy to see that for y £ Y and 
g £ X(# , U) with y = tt^), y £ F if and only if f/y = {gH^g-^y. In 
the sequel we prove that Y has the desired properties. 

Step 2. We prove that for any y £ Y and g £ X(H ,U) with 
y = 7r(g), there is a neighborhood Q of g in G such that 

7r _1 (y)nfi C X(H Q ,U). 

If not, then there is a sequence {g n } C 7r -1 (F) \X(Ho, U) with gf„ — g. 
By the Baire category theorem, there is a compact set K C X(H , U) 
such that ir(K) contains a nonempty open subset of Y. In view of the 
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minimality of the A-action on Y and the left A-invariance of X(Hq, U), 
we may assume that ir(K) contains a neighborhood of y in Y. Since 
Y 3 Tr(g n ) ~~ Hi we ma Y assume that n(g n ) G ir(K). So there are 
sequences {k n } C K and {7^} C T such that g n = k n ^ n . Since K is 
compact, we may also assume that k n — > k for some k G if. Hence 
7n - ► 7, where 7 = k _1 g G I\ It follows that 7™ = 7 for all large n. 
Since ir(g) = n(k) = y G Y , we have 

(gHog-^y = Uy~ = {kH,k~ l )y. 

This implies that 7 G Nq{Hq). Thus for all large n we have 

g n = k nl G X(ii , U)N G (H ) = X(H , U), 

a contradiction. 

Step 3. Let V = ©fc™ G A fc 0' an d V be the projective space of 
V. Consider the representation p = A Ad : G —> GL(V) and 

the corresponding projective representation p : G — > PGL(V"). If S 
is a Lie subgroup of G, we denote ps = f\ im Lie (5), which is a line 
in /\ dimS Q, hence an element in V. By Ratner's theorem, for every 
x G G/F, there is a connected closed subgroup S of G containing U 
such that Ux = Sx. This defines a map 

: G/r 7, = p s . 

Note that if Ux — Sx, then U(ax) = aUx = (aSa~ 1 )(ax). This means 
that (p is A-equivariant, i.e., 

<p(ax) = p(a)cp(x), Va G A,x G G/r. 

We prove below that the map tp is continuous on Y . 

Let y„,,y G Yo with y n — > y. We need to show that <p(y n ) — >• viy)- 
Choose g G X(H ,U) with y = ir(g), and choose g n G G with y n = 
n(g n ) and g n — > g. By Step 2, we may assume that g n G X(H , U) for 
every n. Since y n , y G Y , we have 

Cfyn = {gnH g~ l )y n and Efy = (gH g~ l )y. 

This means that 

= p(gn)pH and = p(g)p Ho - 

Hence ^(y n ) -> • 

Step 4. We prove that there is a connected closed subgroup H of G 
which contains £/ and is normalized by A such that for every y G Yq, we 
have C/y = iiy. Note that a connected subgroup S of G is normalized 
by A if and only if ps is fixed by p(A). Thus it suffices to prove that 
the map <p is constant on Y and <p(Y ) C where V^ 4 is the set of 
p(A)-fixed points in V". 
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Let y G Yo. If f(y) ^ V A , then there is a neighborhood N of (p(y) in 
V such that Nf)V A = ®. Let 

A = {a G A : ay G ^(iV)}. 
Since tp is A-equivariant, we have p(A )ip(y) C A 7 ". Hence 

p(A )^(y) n V A = 0. 

On the other hand, since the p(A)-action on V is IR-diagonalizable, 
there is a finite set \& of homomorphisms A — >■ R* and a direct sum 
decomposition 

V = (J) ^x> where V x = {v G V : p(a)v = x{a)v,Va G A}. 

Note that V A is equal to the union of the projective spaces of V x . 
Choose a nonzero vector v G (f(y), and write v = ^2 xe yrV x , wnere 
/ $' C f and ^ v x G V x for every \ £ ^' ■ Since <p is continuous 
on Yq, <p _1 (A r ) contains a neighborhood of y in Y~o- By the minimality of 
the A-action on Y, A is syndetic in A, i.e., there is a compact subset 
C C A with A C = A. From this it is easy to see that there exist 
Xo ^ ^' an d a sequence {a„} C A such that (Xo 1 x)( a «) — f° r an Y 
X G n {Xo}- Thus 

Xo«TVK> = v X0 + (Xo^KK v X0 . 

This means that p(a n )ip(y) converges to a line in V xo , which is a point 
in V A . Thus p(A )<p(y)C\ V A ^ 0. This is a contradiction, hence proves 
that (p(Y ) C 

Since <p is continuous on Yo and is A-equivariant, and every A-orbit 
in Yq is dense in Yq, it follows that every p(y4)-orbit in (p(Y ), which is 
a single point, is dense in (f(Y ). Thus <p(Y~ ) consists of only one point. 
Hence <p is constant on Yq- 

Step 5. We prove that H is reductive. For a Lie subgroup S of G, 
denote 

Nh(S) = {g G N G jS) : det Ad(p)| Lic(s) = 1}. 

We first prove that A C Nq(H). Consider the character \ : A — y M* 
defined by x( a ) — det Ad(a)| Lie ,^. Let be a nonzero vector in p^. 
Then p(a)p^ = x(a)p^ for every a G A Let g G X(H ,U) with 
7r(g) G Yq. Then = gH^g' 1 , and hence p(g~ l )Y>u G p# . Since Y" is 
compact, there is a compact subset M C G such that A7r(<7) C Y" C 
Mvrfe). This implies A C MFg* 1 . Thus 

x(A) P5 = P {A) V5 c P (Mr( 7 - 1 )p # . 
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By jDMl Thm. 3.4], p(Yg ^p^ is discrete. So p(MYg 1 )p^- is a closed 
subset of V and does not contain 0. Hence ^ x{A)pjj- This implies 
that x(A) ^ M*. So x is trivial. Hence A G N G (H). 

Let W be the unipotent radical of the Zariski closure of H in G. Since 
A C N G (H), we have W G H and A C N G (W). Since the real algebraic 
group AW is solvable and IR-split, by Borel's fixed point theorem [Bj 
Prop. 15.2], AW is contained in a minimal parabolic subgroup P of G. 
The set $ + = {a G $ : ?7 Q C P} is a system of positive roots. So there 
exists a E A such that a(a ) > 1 for every a G $ + . If IV is nontrivial, 
then all eigenvalues of Ad(a )|Lie(vy) are of the form a(a ) (a G $ + ), 
and hence det Ad(ao)|Lie(VK) > 1- This contradicts A C N G (W). So W 
is trivial. Hence if is reductive. The proof of (i) is completed. 

Step 6. We now prove (ii). Since H is reductive, so is L (see e.g. 
[LRl Lem 1.1]), and we have L = N^(H)°. Let y G Y , g G X(fT , ^) 
with 7i (g) G Yo- Then 

Ly = N 1 G (H) y = gN 1 G (H )°n(e). 

By |DMl Thm. 3.4], N G (H )7r(e) is closed. So is also closed. Since 
A C L, we have Y" = Ay G Ly. It remains to prove that is of 
finite volume. If not, then by Proposition [TJ there is a nontrivial IR- 
diagonalizable connected algebraic central subgroup Z of L such that 
Zy is a divergent orbit in G/Y. Since L contains A and is reductive, we 
must have Z G A. Thus Zy C Y\ This contradicts the compactness of 
Y\ □ 

Proof of Theorem Firstly, we remark that if S is a Lie subgroup of G 
containing A, then S has no nontrivial connected unipotent subgroups 
if and only S° G Zq{A). In fact, since S contains A, we have 

Lie(S) = (Lie(S) n Ue{Z G {A))) © 0(Lie(5) n u Q ). 

If S has no nontrivial connected unipotent subgroups, then for every 
a G $ we have Lie(S') flu a = 0. Thus Lie(S') C Lie(Za(A)), and hence 
S° G Zg{A). Conversely, since Zg{A) consists of semisimple elements, 
if S° G Z G (A) then S has no nontrivial connected unipotent subgroups. 

We now prove that Stab^Y") has no nontrivial connected unipotent 
subgroups by induction on dimG. If dimG = 0, there is nothing to 
prove. Assume dim G > and the assertion holds for groups of smaller 
dimension. We first prove the following: 

Claim. If there exists a nontrivial connected normal algebraic sub- 
group N of G such that vr(r) is a lattice in G/N, where tt : G — > G/N is 
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the projection, then any connected unipotent subgroup U of Stabc(y) 
is contained in N. 

Let G' = G/N. Then G' is a reductive real algebraic group, and 
tt(A) is a maximal connected M-diagonalizable subgroup of G' . Let 
7f : G/Y — > G'/ir(Y) be the induced projection. Then tt(Y) is a compact 
7r( A) -minimal subset of G'/ir(Y), and 7r(Z7") is a connected unipotent 
subgroup of Stab G /(7f (Y)). By the induction hypothesis, n(U) is trivial. 
Hence U G N. This proves the claim. 

We proceed the proof by considering three cases. 
Case 1. Suppose G is not semisimple. By |Rag Cor. 8.27], the 



group N = C(G)° satisfies the assumption of the claim. So any con- 
nected unipotent subgroup U of Stabc(^) is contained in C(G). But 
C(G) consists of semisimple elements. So U must be trivial. 

Case 2. Suppose G is semisimple and V is reducible. Then there 
are nontrivial connected normal subgroups G\, Gi of G such that G = 
G1G2, Gi n G2 is discrete, and the assumption of the claim is satisfied 
for N = Gi, G 2 . Thus any connected unipotent subgroup of Stabc(y) 
is contained in (G\ fl G2) , which is trivial. 

Case 3. Suppose G is semisimple and T is irreducible. Suppose, 
to the contrary, that Stabc(Y) has a nontrivial connected unipotent 
subgroup U. Since Stabc(Y) contains A, we may assume that U is 
normalized by A. By Proposition [SX there exist y EY and a connected 
reductive subgroup H of G containing U such that Uy = Hy, and such 
that Ly is closed of finite volume and contains Y, where L = Ng(H)° 
is a reductive real algebraic group containing AH. There are three 
subcases: 

(i) L 7^ G. The fact U C Stab^F) contradicts the induction 
hypothesis. 

(ii) L = G but H G. Then if is a proper normal subgroup of G, 
and has a closed orbit in G/Y of finite volume. This contradicts 
the irreducibility of Y. 

(iii) H = G. Then Y D Uy = Hy = G/Y. Hence A acts minimally 
on G/Y. But by |PP4 Thm. 2.8], if C is Cartan subgroup of 
G containing A, then there are compact C-orbits in G/Y, a 
contradiction. 

This completes the proof of Theorem □ 

4. Proof of Theorem [3] 

The root system $ may be non-reduced. For a G <£, denote [a] = 
{ca : c > 0} fl $. Then [a] has three possibilities: {a}, {a, 2a}, 
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and {a, a/2}. Let U[ a ] = J2pe{ a ] u /?' an< ^ ^ ^[«] = ex P( u [a]) be the 
unipotent group with Lie algebra U[ a ]- We first prove: 

Lemma 9. Let R C G be a closed subset invariant under the conjuga- 
tion of A such that R n U[ a ] C {e} for every a G $. TTien i/iere exists 
a neighborhood Q of e in G such that RC\Q C Zq(A). 

Proof. Suppose the conclusion of the lemma is not true. Then there 
exists a sequence r n G exp _1 (_R) \ Lie(Zc(v4)) with r n — > 0. Since 
g = Lie(Z G (yl)) © ae<J) u a , we can write 



u 



where z n G Lie(Zc(A)), u nQ G u a . Note that z n and u n>a converge to 
as n — > oo. Identifying a character A — > M* with its differential, we 
can think of a root a as an element in a*. By passing to a subsequence, 
we may assume that there exists ao £ Q such that 

|u n , Q |W < |u n , ao |ra, Va G G N, 

where |u njQ | (resp. \a\) is the norm of u„ )Q , (resp. a) with respect to 

a fixed inner product on q (resp. a*). Since r n £ Lie(Za(A)), we have 

i 

l u n,a l > 0- Let t n G R be such that e tn = \u n ,a \ i Q oi 2 , and let a G a 
be such that a (a) = (a, ccq) for every a G $. Then 



Ad(exp(t n a))r n = z n + £V" a(a) u r 



a£$\{oo} 

By passing to a subsequence, we may assume that 

for some Ui G u Qo with |ui| = 1. For a G $ \ {«o}, we have 



( a ."0) l"ll"o|-("."Q> 

lii an 1 2 lii I <T I n | Q()l 2 



If a G $ s [ao], then (a, a ) < |o;||q;o|, and hence 

( a ."o> 

If [a ] 7^ {«o} an d a ^ [ a o] \ { a o} ; then (a, a ) = |a||a |, and we have 

1".°0> 



u 



Q!()l 2 111 1^1 



n,ao I I "ra,a: 



REMARKS ON MINIMAL SETS AND A CONJECTURE OF MARGULIS 13 



Hence by passing to a further subsequence, we may assume that 

(°.°o) 

|u niC J l"oF u n , Q ->■ u 2 

for some u 2 G u a . In summary, a subsequence of Ad(exp(t n a))r n con- 
verges to a nonzero element u G U[ ao ], where u = u x if [a ] = {«()}> 
and u = ui + u 2 if [ao] 7^ {«o}- Since exp _1 (i?) is closed and Ad(A)- 
invariant, we have u G exp~ 1 (_R). Thus e 7^ exp(u) £ i?fl ^[oo]j a 
contradiction. □ 

Lemma 10. Let Y G G/T be a compact A-minimal set, a G $, and 
Y a G Y be an A a -minimal set. Suppose AY a = Y . Then Stab^(Y a ) is 
cocompact in A. 

Proof. We first prove that for any open subset B of A, BY a is open 
in Y . It suffices to prove that for every b G B, bY a is contained in 
the interior int(JBy^) of BY a . Let C be a compact neighborhood of 
e in A such that bC~ l C G B, and let {a n } be a sequence in A such 
that U^Li fl nC = A. Then UnLi ^Cl^ = F- By the Baire category 
theorem, some a n CY a , and hence CY a , has an interior point. Let c G C 
be such that int(CY Q ) n cY a 7^ 0. Then 

int(Sy a ) n bY a D int(6c _1 Cy a ) n 6F a = 6c _1 (int(Cr a ) n cF a ) 7^ 0. 

Since int(Sl^) is open A^-invariant and bY a is Ac-minimal, we have 
bY a G int(BY a ). Thus BY a is open in Y. 

Now we prove that Stab^Ya) is cocompact in A. Since Stab^F*) G> 
A a , it suffices to prove that Stab^F*) 7^ A a . Let B± G B 2 G ■ ■ ■ be a 
nested sequence of bounded open subsets of A such that U^Li Bn = A. 
Then {B n Y a } is an open cover of Y. Since F is compact, there exists 
n GN such that Y = B no Y a . Let a £ Abe such that a~ 1 B no C\A a = 0. 
Since aY^ is A^-minimal and Y = B no Y a , there exists b G -B no with 
aY a = bY a . It follows that a" 1 ^ G Stab^(Y„). But a~ l b ^ A a . So 
StabA(Y a ) 7^ A Q . This proves the lemma. □ 

We are now prepared to prove Theorem [3J 

Proof of Theorem 0. The assumption implies that every Aa-minimal 
subset of Y is of the form aY a (a G A), hence by Lemma [Tu] has a 
cocompact stabilizer in A. Let 

R= { g g G : gYHY ^ 0}. 

Then i? is closed and invariant under the conjugation of A. We first 
prove that RClU[ a ] = {e} for every a G $. Suppose the contradiction. 
Then there exists e 7^ w G C/[ a ] such that uYC\Y 7^ 0. Since u commutes 
with A a , the compact set uY D F is A a -invariant, hence contains an 
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A Q -minimal set Y' . Since Statu (V') is cocompact in A, there exists a 
sequence {a n } C Stab^F') such that a n ua~ l — >• e. Note that u~ l Y' C 
Y" is also A Q -minimal. So u Y' = aY' for some a G A, i.e., ua G 
Stab G (F'). It follows that 

Stably') 3 (uG^a^im) -1 ^ 1 = u(a n ua~ l )~ l — > u. 

By the closedness of Stab^F'), we have u G Stab(3(F'). So a^ua" 1 G 
Stabc(y)- This implies that the group U[ a -\r\Stabc{Y') is non-discrete. 

So U == (Z7[ a ] fl Stabc(y)) * s nontrivial. On the other hand, since U[ a ] 
and Stabc(F') are normalized by Stab^F'), so is U. But Nq{U) is 
Zariski closed in G and Stab^F') is Zariski dense in A. So A normalizes 
U. Hence 

UY = U AY' = AUY' = AY' = Y. 

It follows that U C StabctX). This conflicts Theorem [5] Hence R(l 
U[o\ = {e} for every a G $. 

Let Z = Zg(A)°. Then Z is the direct product of A and a connected 
compact subgroup M of G centralized by A. Let y G V. We claim 
that is compact and contains Y. By Lemma and the preceding 
paragraph, there exists an open neighborhood Q of e in G such that 
Ron C Z. It follows that FnQi/C (i? D C Zy. Since F is 
A-minimal and F fl f2y is a neighborhood of y in F, we have F = 
A(Y n fh/) C Zy. This in turn implies that Zy = ZY = MY is 
compact, proving the claim. 

Let A = {g G Z : gy = y}. Then A is a lattice in Z and the natural 
map Z/A — > Zy is a homeomorphism. Let p:Z = AxM->Mbe the 
projection, and let T = (p(A))°. By [Raj Thm. 8.24], T is solvable, 
hence is a compact torus. Note that AT = (Ap(A))° = (AA)°. So ATy 
is closed and contains Ay as a dense subset. It follows that F = ATy. 
This completes the proof. □ 
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